Skip to content

Add shutdown outcome telemetry to LLM queue worker - #2746

Merged
Chris0Jeky merged 3 commits into
mainfrom
issue-1632/shutdown-metrics
Sep 6, 2026
Merged

Add shutdown outcome telemetry to LLM queue worker#2746
Chris0Jeky merged 3 commits into
mainfrom
issue-1632/shutdown-metrics

Conversation

@Chris0Jeky

Copy link
Copy Markdown
Owner

Summary

  • Emit finite abandoned_shutdown and cancelled_requeued worker outcomes only after the corresponding shutdown write succeeds.
  • Cover processing cancellation, retry-backoff cancellation, and failed-cleanup negative controls.

Verification

  • dotnet restore backend/tests/Taskdeck.Api.Tests/Taskdeck.Api.Tests.csproj
  • dotnet test backend/tests/Taskdeck.Api.Tests/Taskdeck.Api.Tests.csproj -c Release -m:1 --filter "FullyQualifiedName~LlmQueueToProposalWorkerTests" --no-restore (52 passed)
  • git diff --check

No merge or issue close performed. Refs #1632

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Chris0Jeky and others added 2 commits September 6, 2026 12:52
…hutdown, shutdown_release_failed); telemetry listener filtered by worker, synchronized, and run in a non-parallel collection (review HIGH-1, MEDIUM-2)
@Chris0Jeky

Copy link
Copy Markdown
Owner Author

Fresh-context review record (coordinator, 2026-09-06; read-only Opus 5 reviewer over the diff against the merge base; route: the standing D-12 rule for Codex-session PRs — review, fix MEDIUM+ in-PR, merge, disposition comment).

VERDICT: FIX-FIRST (round 1) → fixed (round 2, coordinator commit 2f6531cea; LlmQueueToProposalWorkerTests plus WorkerResilienceTests 67/67 locally). HIGH-1 — the test helper listened to the process-global WorkerItemsProcessed counter, filtered only on the outcome tag, appended to an unsynchronised List<string> and asserted exact single-element sequences, while TranscriptTriageWorkerTests, the webhook worker tests and both WorkerResilienceTests classes drive the same counter in parallel collections: intermittent reds on unrelated PRs and a data race. The test class now sits in a DisableParallelization collection, the listener filters on worker.name and locks the list. MEDIUM-2abandoned_shutdown was emitted exactly when the row was successfully released (nothing abandoned) and nothing at all when the release write failed; the outcomes are now released_on_shutdown and shutdown_release_failed, the latter recorded only from the release write's catch (a row that had already moved on emits nothing, which the retry-backoff cancellation test pins). Recorded, not fixed (LOW): RecordWorkerOutcome sits inside the requeue try, so a throwing meter listener after a committed requeue logs the opposite of the persisted state; CompleteRetryTransitionOnShutdownAsync returns true on a no-op transition, so cancelled_requeued can count a requeue that did not occur; the second negative control asserts NotContain where BeEmpty would be the real control. Confirmed clean: the #2620 cooperative-stop contract, tag cardinality, no ids or content in tags, layer purity.

@Chris0Jeky
Chris0Jeky merged commit 0b56a12 into main Sep 6, 2026
35 checks passed
@Chris0Jeky
Chris0Jeky deleted the issue-1632/shutdown-metrics branch September 6, 2026 12:31
@github-project-automation github-project-automation Bot moved this from Pending to Done in Taskdeck Execution Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant